{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ChatGLM3 Lora 实战"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step1 导入相关包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\Miniconda\\envs\\geo\\lib\\site-packages\\numpy\\_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:\n",
      "d:\\Miniconda\\envs\\geo\\lib\\site-packages\\numpy\\.libs\\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll\n",
      "d:\\Miniconda\\envs\\geo\\lib\\site-packages\\numpy\\.libs\\libopenblas64__v0.3.21-gcc_10_3_0.dll\n",
      "  warnings.warn(\"loaded more than 1 DLL from .libs:\"\n"
     ]
    }
   ],
   "source": [
    "from datasets import Dataset,load_dataset\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, DataCollatorForSeq2Seq, TrainingArguments, Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = '/data/datasets/alpaca_data_zh/alpaca_gpt4_data_zh.json'\n",
    "pretrain_model_dir = \"/data/models/modelscope/ZhipuAI/chatglm3-6b\"\n",
    "save_dir = '/data/logs/chatglm3_1b4_lora_tuning'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step2 加载数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset({\n",
      "    features: ['instruction', 'output', 'input'],\n",
      "    num_rows: 9763\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "datasets = load_dataset('json', data_files=data_dir,split='train')\n",
    "datasets = datasets.train_test_split(test_size=0.8)\n",
    "ds = datasets[\"train\"]\n",
    "print(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'instruction': ['生成一个有关如何使演示更具吸引力的提示列表。',\n",
       "  '描述你认为2030年互联网会是什么样子。',\n",
       "  '模拟两个人之间的对话，A人在家具店工作，B人来店里参观。'],\n",
       " 'output': ['以下是一些如何使演示更具吸引力的提示：\\n\\n1. 采用引人入胜的开场白：开场白是吸引听众注意力的重要部分，一开始就用一段引人入胜的开场白会让听众对你的演示产生兴趣。\\n\\n2. 使用视觉辅助工具：视觉辅助工具如图片、图表和视频能够强调重点，帮助听众更好地理解演示的内容。\\n\\n3. 采用清晰的结构：演示的结构应该清晰明了，使听众容易理解并跟随演示的进程。\\n\\n4. 打破单调：尝试在演示中使用幽默、轻松的语调和肢体语言，以打破单调并吸引听众的注意力。\\n\\n5. 与听众互动：通过提问、讨论和小组活动等方式与听众互动，可以使演示更具吸引力。\\n\\n6. 注意语速和音量：语速和音量应适中，语言清晰流畅，有助于听众更好地理解演示的内容。\\n\\n7. 鼓励反馈：鼓励听众在演示的过程中提供反馈，可以帮助你更好地调整演示的内容和方式。\\n\\n8. 结束语：在结束演示时使用一个令人难忘的结束语，可以帮助听众记住你的演示内容。',\n",
       "  '到2030年，互联网的演变将继续加速，且会发生重大改变。以下是一些可能的情景：\\n\\n1. 5G 及更先进的网络技术： 5G网络的覆盖范围将更广，而且响应速度更快。这将为我们提供更快速、更稳定、更高质量的网络连接体验。\\n\\n2. 物联网： 智能家居设备以及更多与互联网连接的设备将普及。智能家居、智能健康等物联网应用将更加便捷和完善。\\n\\n3. 增强现实和虚拟现实： 到2030年，增强现实和虚拟现实技术将更加成熟，这将改变我们的娱乐方式。人们将能够在虚拟世界中体验前所未有的现实感。\\n\\n4. 人工智能： 到2030年，人工智能将更加成熟，深度学习和自然语言处理等技术将更加先进。我们将能够与智能助手进行更加自然的交流。\\n\\n5. 更多的个性化和智能化： 互联网公司将使用更多的大数据分析技术，为用户提供更加个性化和智能化的服务。\\n\\n6. 更严格的数据保护： 随着人们对隐私日益重视，政府将会出台更严格的法规来保护用户的数据。\\n\\n总之，到2030年，我们将会有一个更快速、更智能、更个性化、更安全的互联网体验。',\n",
       "  'A: 欢迎光临我们的家具店！你好，我是销售人员A，需要我为您提供一些帮助吗？\\n\\nB: 嗨，你好。我正在寻找一张餐桌和一些餐椅。\\n\\nA: 当然，我们店里有很多不同的款式和尺寸供您选择。您希望购买什么类型的餐桌和餐椅呢？\\n\\nB: 我喜欢木质的，希望桌子能容纳6到8个人，椅子要舒适。\\n\\nA: 非常好。那我带您去看一看我们的木质餐桌和椅子区域。这一款是我们最畅销的，桌子采用实木制作，非常耐用。同时，椅子也是木质的，带有软垫，坐起来很舒适。\\n\\nB: 这款看起来不错，能容纳几个人？\\n\\nA: 这张桌子的长度可以容纳8个人坐在一起。\\n\\nB: 非常好。我还想看看其他的颜色和款式。\\n\\nA: 当然，我们还有其他颜色和款式的餐桌和餐椅供您选择。这边请，我带您去看看。'],\n",
       " 'input': ['', '', '']}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step3 数据集预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatGLMTokenizer(name_or_path='/data/models/modelscope/ZhipuAI/chatglm3-6b', vocab_size=64798, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side='left', truncation_side='right', special_tokens={'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={\n",
       "\t\n",
       "}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(pretrain_model_dir, trust_remote_code=True)\n",
    "tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'input_ids': [64790, 64792, 2893, 30917, 30994], 'attention_mask': [1, 1, 1, 1, 1], 'position_ids': [0, 1, 2, 3, 4]},\n",
       " 2)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(tokenizer.eos_token), tokenizer.eos_token_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# [gMASK]sop<|user|> \\n Prompt<|assistant|> \\n Response eos_token\n",
    "def process_func(example):\n",
    "    MAX_LENGTH = 256\n",
    "    input_ids, attention_mask, labels = [], [], []\n",
    "    instruction = \"\\n\".join([example[\"instruction\"], example[\"input\"]]).strip()     # query\n",
    "    instruction = tokenizer.build_chat_input(instruction, history=[], role=\"user\")  # [gMASK]sop<|user|> \\n query<|assistant|>\n",
    "    response = tokenizer(\"\\n\" + example[\"output\"], add_special_tokens=False)        # \\n response, 缺少eos token\n",
    "    input_ids = instruction[\"input_ids\"][0].numpy().tolist() + response[\"input_ids\"] + [tokenizer.eos_token_id]\n",
    "    attention_mask = instruction[\"attention_mask\"][0].numpy().tolist() + response[\"attention_mask\"] + [1]\n",
    "    labels = [-100] * len(instruction[\"input_ids\"][0].numpy().tolist()) + response[\"input_ids\"] + [tokenizer.eos_token_id]\n",
    "    if len(input_ids) > MAX_LENGTH:\n",
    "        input_ids = input_ids[:MAX_LENGTH]\n",
    "        attention_mask = attention_mask[:MAX_LENGTH]\n",
    "        labels = labels[:MAX_LENGTH]\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"attention_mask\": attention_mask,\n",
    "        \"labels\": labels\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "25215983a22742ad9bf6e192c15bc314",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/9763 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['input_ids', 'attention_mask', 'labels'],\n",
       "    num_rows: 9763\n",
       "})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_ds = ds.map(process_func, remove_columns=ds.column_names)\n",
    "tokenized_ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[gMASK]sop<|user|> \\n 描述你认为2030年互联网会是什么样子。<|assistant|> \\n到2030年，互联网的演变将继续加速，且会发生重大改变。以下是一些可能的情景：\\n\\n1. 5G 及更先进的网络技术： 5G网络的覆盖范围将更广，而且响应速度更快。这将为我们提供更快速、更稳定、更高质量的网络连接体验。\\n\\n2. 物联网： 智能家居设备以及更多与互联网连接的设备将普及。智能家居、智能健康等物联网应用将更加便捷和完善。\\n\\n3. 增强现实和虚拟现实： 到2030年，增强现实和虚拟现实技术将更加成熟，这将改变我们的娱乐方式。人们将能够在虚拟世界中体验前所未有的现实感。\\n\\n4. 人工智能： 到2030年，人工智能将更加成熟，深度学习和自然语言处理等技术将更加先进。我们将能够与智能助手进行更加自然的交流。\\n\\n5. 更多的个性化和智能化： 互联网公司将使用更多的大数据分析技术，为用户提供更加个性化和智能化的服务。\\n\\n6. 更严格的数据保护： 随着人们对隐私日益重视，政府将会出台更严格的法规'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(tokenized_ds[1][\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n到2030年，互联网的演变将继续加速，且会发生重大改变。以下是一些可能的情景：\\n\\n1. 5G 及更先进的网络技术： 5G网络的覆盖范围将更广，而且响应速度更快。这将为我们提供更快速、更稳定、更高质量的网络连接体验。\\n\\n2. 物联网： 智能家居设备以及更多与互联网连接的设备将普及。智能家居、智能健康等物联网应用将更加便捷和完善。\\n\\n3. 增强现实和虚拟现实： 到2030年，增强现实和虚拟现实技术将更加成熟，这将改变我们的娱乐方式。人们将能够在虚拟世界中体验前所未有的现实感。\\n\\n4. 人工智能： 到2030年，人工智能将更加成熟，深度学习和自然语言处理等技术将更加先进。我们将能够与智能助手进行更加自然的交流。\\n\\n5. 更多的个性化和智能化： 互联网公司将使用更多的大数据分析技术，为用户提供更加个性化和智能化的服务。\\n\\n6. 更严格的数据保护： 随着人们对隐私日益重视，政府将会出台更严格的法规'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(list(filter(lambda x: x != -100, tokenized_ds[1][\"labels\"])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step4 创建模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.\n",
      "`low_cpu_mem_usage` was None, now set to True since model is quantized.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bin d:\\Miniconda\\envs\\geo\\lib\\site-packages\\bitsandbytes\\libbitsandbytes_cuda116.dll\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85768b2fc57f4c9abd690860c08f1f89",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/7 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "\"\"\"\n",
    "新版本中需要将modeling_chatglm源码中的613行部分进行调整，代码如下：\n",
    "\n",
    "```\n",
    "if not kv_caches:\n",
    "    kv_caches = [None for _ in range(self.num_layers)]\n",
    "else:\n",
    "    kv_caches = kv_caches[1]\n",
    "```\n",
    "\n",
    "如果不进行调整，后续chat阶段会报错\n",
    "\"\"\"\n",
    "# 多卡情况，可以去掉device_map=\"auto\"，否则会将模型拆开\n",
    "model = AutoModelForCausalLM.from_pretrained(pretrain_model_dir, trust_remote_code=True, torch_dtype=torch.half,load_in_8bit=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformer.embedding.word_embeddings.weight\n",
      "transformer.encoder.layers.0.input_layernorm.weight\n",
      "transformer.encoder.layers.0.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.0.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.0.self_attention.dense.weight\n",
      "transformer.encoder.layers.0.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.0.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.0.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.1.input_layernorm.weight\n",
      "transformer.encoder.layers.1.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.1.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.1.self_attention.dense.weight\n",
      "transformer.encoder.layers.1.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.1.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.1.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.2.input_layernorm.weight\n",
      "transformer.encoder.layers.2.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.2.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.2.self_attention.dense.weight\n",
      "transformer.encoder.layers.2.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.2.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.2.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.3.input_layernorm.weight\n",
      "transformer.encoder.layers.3.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.3.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.3.self_attention.dense.weight\n",
      "transformer.encoder.layers.3.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.3.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.3.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.4.input_layernorm.weight\n",
      "transformer.encoder.layers.4.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.4.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.4.self_attention.dense.weight\n",
      "transformer.encoder.layers.4.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.4.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.4.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.5.input_layernorm.weight\n",
      "transformer.encoder.layers.5.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.5.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.5.self_attention.dense.weight\n",
      "transformer.encoder.layers.5.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.5.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.5.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.6.input_layernorm.weight\n",
      "transformer.encoder.layers.6.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.6.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.6.self_attention.dense.weight\n",
      "transformer.encoder.layers.6.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.6.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.6.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.7.input_layernorm.weight\n",
      "transformer.encoder.layers.7.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.7.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.7.self_attention.dense.weight\n",
      "transformer.encoder.layers.7.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.7.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.7.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.8.input_layernorm.weight\n",
      "transformer.encoder.layers.8.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.8.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.8.self_attention.dense.weight\n",
      "transformer.encoder.layers.8.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.8.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.8.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.9.input_layernorm.weight\n",
      "transformer.encoder.layers.9.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.9.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.9.self_attention.dense.weight\n",
      "transformer.encoder.layers.9.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.9.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.9.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.10.input_layernorm.weight\n",
      "transformer.encoder.layers.10.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.10.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.10.self_attention.dense.weight\n",
      "transformer.encoder.layers.10.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.10.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.10.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.11.input_layernorm.weight\n",
      "transformer.encoder.layers.11.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.11.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.11.self_attention.dense.weight\n",
      "transformer.encoder.layers.11.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.11.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.11.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.12.input_layernorm.weight\n",
      "transformer.encoder.layers.12.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.12.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.12.self_attention.dense.weight\n",
      "transformer.encoder.layers.12.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.12.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.12.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.13.input_layernorm.weight\n",
      "transformer.encoder.layers.13.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.13.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.13.self_attention.dense.weight\n",
      "transformer.encoder.layers.13.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.13.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.13.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.14.input_layernorm.weight\n",
      "transformer.encoder.layers.14.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.14.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.14.self_attention.dense.weight\n",
      "transformer.encoder.layers.14.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.14.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.14.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.15.input_layernorm.weight\n",
      "transformer.encoder.layers.15.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.15.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.15.self_attention.dense.weight\n",
      "transformer.encoder.layers.15.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.15.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.15.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.16.input_layernorm.weight\n",
      "transformer.encoder.layers.16.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.16.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.16.self_attention.dense.weight\n",
      "transformer.encoder.layers.16.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.16.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.16.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.17.input_layernorm.weight\n",
      "transformer.encoder.layers.17.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.17.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.17.self_attention.dense.weight\n",
      "transformer.encoder.layers.17.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.17.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.17.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.18.input_layernorm.weight\n",
      "transformer.encoder.layers.18.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.18.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.18.self_attention.dense.weight\n",
      "transformer.encoder.layers.18.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.18.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.18.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.19.input_layernorm.weight\n",
      "transformer.encoder.layers.19.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.19.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.19.self_attention.dense.weight\n",
      "transformer.encoder.layers.19.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.19.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.19.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.20.input_layernorm.weight\n",
      "transformer.encoder.layers.20.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.20.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.20.self_attention.dense.weight\n",
      "transformer.encoder.layers.20.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.20.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.20.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.21.input_layernorm.weight\n",
      "transformer.encoder.layers.21.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.21.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.21.self_attention.dense.weight\n",
      "transformer.encoder.layers.21.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.21.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.21.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.22.input_layernorm.weight\n",
      "transformer.encoder.layers.22.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.22.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.22.self_attention.dense.weight\n",
      "transformer.encoder.layers.22.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.22.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.22.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.23.input_layernorm.weight\n",
      "transformer.encoder.layers.23.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.23.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.23.self_attention.dense.weight\n",
      "transformer.encoder.layers.23.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.23.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.23.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.24.input_layernorm.weight\n",
      "transformer.encoder.layers.24.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.24.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.24.self_attention.dense.weight\n",
      "transformer.encoder.layers.24.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.24.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.24.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.25.input_layernorm.weight\n",
      "transformer.encoder.layers.25.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.25.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.25.self_attention.dense.weight\n",
      "transformer.encoder.layers.25.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.25.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.25.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.26.input_layernorm.weight\n",
      "transformer.encoder.layers.26.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.26.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.26.self_attention.dense.weight\n",
      "transformer.encoder.layers.26.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.26.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.26.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.layers.27.input_layernorm.weight\n",
      "transformer.encoder.layers.27.self_attention.query_key_value.weight\n",
      "transformer.encoder.layers.27.self_attention.query_key_value.bias\n",
      "transformer.encoder.layers.27.self_attention.dense.weight\n",
      "transformer.encoder.layers.27.post_attention_layernorm.weight\n",
      "transformer.encoder.layers.27.mlp.dense_h_to_4h.weight\n",
      "transformer.encoder.layers.27.mlp.dense_4h_to_h.weight\n",
      "transformer.encoder.final_layernorm.weight\n",
      "transformer.output_layer.weight\n"
     ]
    }
   ],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    print(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PEFT Step1 配置文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LoraConfig(peft_type=<PeftType.LORA: 'LORA'>, auto_mapping=None, base_model_name_or_path=None, revision=None, task_type=<TaskType.CAUSAL_LM: 'CAUSAL_LM'>, inference_mode=False, r=8, target_modules=None, lora_alpha=8, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={}, megatron_config=None, megatron_core='megatron.core', loftq_config={}, use_dora=False, layer_replication=None, runtime_config=LoraRuntimeConfig(ephemeral_gpu_offload=False))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from peft import LoraConfig, TaskType, get_peft_model, PeftModel\n",
    "\n",
    "config = LoraConfig(task_type=TaskType.CAUSAL_LM,)\n",
    "config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PEFT Step2 创建模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = get_peft_model(model, config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LoraConfig(peft_type=<PeftType.LORA: 'LORA'>, auto_mapping=None, base_model_name_or_path=None, revision=None, task_type=<TaskType.CAUSAL_LM: 'CAUSAL_LM'>, inference_mode=False, r=8, target_modules={'query_key_value'}, lora_alpha=8, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={}, megatron_config=None, megatron_core='megatron.core', loftq_config={}, use_dora=False, layer_replication=None, runtime_config=LoraRuntimeConfig(ephemeral_gpu_offload=False))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "base_model.model.transformer.embedding.word_embeddings.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.0.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.0.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.0.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.0.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.0.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.0.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.0.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.0.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.0.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.0.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.1.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.1.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.1.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.1.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.1.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.1.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.1.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.1.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.1.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.1.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.2.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.2.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.2.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.2.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.2.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.2.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.2.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.2.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.2.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.2.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.3.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.3.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.3.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.3.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.3.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.3.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.3.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.3.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.3.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.3.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.4.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.4.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.4.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.4.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.4.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.4.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.4.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.4.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.4.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.4.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.5.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.5.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.5.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.5.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.5.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.5.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.5.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.5.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.5.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.5.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.6.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.6.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.6.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.6.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.6.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.6.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.6.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.6.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.6.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.6.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.7.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.7.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.7.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.7.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.7.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.7.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.7.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.7.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.7.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.7.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.8.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.8.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.8.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.8.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.8.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.8.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.8.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.8.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.8.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.8.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.9.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.9.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.9.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.9.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.9.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.9.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.9.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.9.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.9.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.9.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.10.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.10.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.10.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.10.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.10.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.10.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.10.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.10.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.10.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.10.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.11.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.11.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.11.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.11.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.11.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.11.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.11.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.11.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.11.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.11.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.12.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.12.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.12.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.12.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.12.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.12.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.12.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.12.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.12.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.12.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.13.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.13.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.13.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.13.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.13.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.13.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.13.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.13.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.13.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.13.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.14.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.14.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.14.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.14.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.14.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.14.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.14.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.14.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.14.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.14.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.15.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.15.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.15.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.15.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.15.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.15.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.15.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.15.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.15.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.15.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.16.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.16.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.16.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.16.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.16.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.16.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.16.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.16.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.16.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.16.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.17.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.17.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.17.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.17.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.17.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.17.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.17.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.17.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.17.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.17.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.18.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.18.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.18.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.18.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.18.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.18.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.18.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.18.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.18.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.18.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.19.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.19.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.19.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.19.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.19.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.19.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.19.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.19.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.19.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.19.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.20.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.20.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.20.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.20.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.20.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.20.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.20.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.20.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.20.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.20.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.21.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.21.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.21.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.21.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.21.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.21.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.21.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.21.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.21.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.21.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.22.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.22.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.22.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.22.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.22.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.22.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.22.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.22.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.22.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.22.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.23.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.23.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.23.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.23.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.23.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.23.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.23.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.23.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.23.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.23.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.24.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.24.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.24.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.24.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.24.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.24.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.24.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.24.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.24.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.24.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.25.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.25.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.25.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.25.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.25.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.25.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.25.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.25.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.25.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.25.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.26.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.26.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.26.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.26.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.26.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.26.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.26.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.26.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.26.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.26.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.27.input_layernorm.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.27.self_attention.query_key_value.base_layer.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.27.self_attention.query_key_value.base_layer.bias torch.float16\n",
      "base_model.model.transformer.encoder.layers.27.self_attention.query_key_value.lora_A.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.27.self_attention.query_key_value.lora_B.default.weight torch.float32\n",
      "base_model.model.transformer.encoder.layers.27.self_attention.dense.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.27.post_attention_layernorm.original_module.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.27.post_attention_layernorm.modules_to_save.default.weight torch.float16\n",
      "base_model.model.transformer.encoder.layers.27.mlp.dense_h_to_4h.weight torch.int8\n",
      "base_model.model.transformer.encoder.layers.27.mlp.dense_4h_to_h.weight torch.int8\n",
      "base_model.model.transformer.encoder.final_layernorm.weight torch.float16\n",
      "base_model.model.transformer.output_layer.weight torch.float16\n"
     ]
    }
   ],
   "source": [
    "for name, parameter in model.named_parameters():\n",
    "    print(name,parameter.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 2,064,384 || all params: 6,245,648,384 || trainable%: 0.0331\n"
     ]
    }
   ],
   "source": [
    "model.print_trainable_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model\n",
    "# model = model.half()\n",
    "# 4、开启梯度检查点时，一定要执行该方法\n",
    "model.enable_input_require_grads() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step5 配置训练参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    output_dir=save_dir,\n",
    "    per_device_train_batch_size=1,\n",
    "    gradient_accumulation_steps=16,\n",
    "    logging_steps=10,\n",
    "    num_train_epochs=1,\n",
    "    learning_rate=1e-4,\n",
    "    remove_unused_columns=False,\n",
    "    save_strategy=\"epoch\",\n",
    "    # adam_epsilon=1e-6,\n",
    "    gradient_checkpointing=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step6 创建训练器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=args,\n",
    "    train_dataset=tokenized_ds.select(range(6000)),\n",
    "    data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step7 模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using an old version of the checkpointing format that is deprecated (We will also silently ignore `gradient_checkpointing_kwargs` in case you passed it).Please update to the new format on your modeling file. To use the new format, you need to completely remove the definition of the method `_set_gradient_checkpointing` in your model.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d887f62ee454b30999528df59e8734a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/375 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 12.763, 'grad_norm': nan, 'learning_rate': 9.733333333333335e-05, 'epoch': 0.03}\n",
      "{'loss': 12.7783, 'grad_norm': nan, 'learning_rate': 9.466666666666667e-05, 'epoch': 0.05}\n",
      "{'loss': 12.8399, 'grad_norm': nan, 'learning_rate': 9.200000000000001e-05, 'epoch': 0.08}\n",
      "{'loss': 12.7784, 'grad_norm': nan, 'learning_rate': 8.933333333333334e-05, 'epoch': 0.11}\n",
      "{'loss': 12.8265, 'grad_norm': nan, 'learning_rate': 8.666666666666667e-05, 'epoch': 0.13}\n",
      "{'loss': 12.855, 'grad_norm': nan, 'learning_rate': 8.4e-05, 'epoch': 0.16}\n"
     ]
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from safetensors import safe_open\n",
    "\n",
    "with safe_open(\"./chatbot/checkpoint-187/adapter_model.safetensors\", framework=\"pt\") as f:\n",
    "    for key in f.keys():\n",
    "        if \".0.post_attention_layernorm\" in key:\n",
    "            print(key)\n",
    "            print(f.get_tensor(key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step8 模型推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "print(model.chat(tokenizer, \"数学考试怎么考高分？\", history=[])[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "transformers",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
